Eyewear device user interface

ABSTRACT

Methods and system for creating and using graphical user interfaces (GUIs) for eyewear devices and eyewear devices including GUIs. The eyewear devices detect finger touches on a surface of the eyewear device and present GUIs on image displays of the eyewear device that are responsive to the finger touches.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 62/905,891 entitled EYEWEAR DEVICE USER INTERFACE, filed on Sep. 25,2019, the contents of which are incorporated fully herein by reference.

TECHNICAL FIELD

The present subject matter relates to electronic devices and, moreparticularly, to a user interface for use with eyewear devices.

BACKGROUND

Computing devices, such as wearable devices, including portable eyeweardevices (e.g., smartglasses, headwear, and headgear), necklaces, andsmartwatches and mobile devices (e.g., tablets, smartphones, andlaptops) integrate image displays and cameras. Graphical user interfaces(GUIs) are used to interact with content displayed on computing devices.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawing figures depict one or more implementations, by way ofexample only, not by way of limitations. In the figures, like referencenumerals refer to the same or similar elements.

FIG. 1A is a right side view of a hardware configuration of an eyeweardevice, which includes an input surface for interacting with a graphicaluser interface (GUI).

FIGS. 1B and 1C are rear views of hardware configurations of the eyeweardevice, including two different types of image displays.

FIG. 1D depicts a schematic view of operation for interaction with theGUI using the input surface of the eyewear.

FIG. 1E is a top cross-sectional view of a left chunk of the eyeweardevice of FIGS. 1B and 1C depicting the input tracker and a circuitboard.

FIGS. 2A, 2B, and 2C depict a schematic view of operation illustrating apress (FIG. 2A), a hold (FIG. 2B), and a release (FIG. 2C) performed onthe input surface.

FIG. 2D depict a schematic view of operation illustrating a swipe leftand a swipe right performed on the input surface.

FIG. 2E depict a schematic view of operation illustrating a swipe up anda swipe down performed on the input surface.

FIGS. 2F and 2G illustrate visual aspects of an example GUI.

FIG. 2H is a schematic view of a remote accessory for controlling a GUIpresented on an eyewear device.

FIG. 3 is a high-level functional block diagram of components of theeyewear device with an input tracker to identify finger touches andgestures along with a mobile device and a server system connected viavarious networks.

FIGS. 4A and 4B are an illustration and a schematic, respectively,showing a GUI for a left-hand interaction on a left side of the eyeweardevice.

FIGS. 4C and 4D are an illustration and a schematic, respectively,showing a GUI for a right-hand interaction on a right side of theeyewear device.

FIG. 4E is an illustration showing a GUI for a left-hand interaction ona left side of the eyewear device for selecting from many choices.

FIG. 4F is an illustration of a GUI for an accessory interaction withthe eyewear device.

FIG. 4G is an illustration showing another GUI for an accessoryinteraction with the eyewear device for selecting from many choices.

FIGS. 5A, 5B, and 5C are illustrations of a GUI interaction for imagesreceived from another user.

FIGS. 5D, 5E, and 5F are illustrations of a GUI interaction forreceiving and responding to messages from another user.

FIGS. 5G, 5H, and 5I are illustrations of a GUI interaction for sendingan image/video to another user.

FIG. 6 is a block diagram of a GUI engine for implementing a GUI on theeyewear device.

FIG. 7A, 7B, and 7C are flow charts for implementing a GUI on theeyewear device.

FIG. 8 is a high-level functional block diagram of a client device thatcommunicates via a network with a server system;

FIG. 9 is a diagrammatic representation of a machine in the form of acomputer system within which a set of instructions may be executed forcausing the machine to perform any one or more of the methodologiesdiscussed herein, in accordance with some examples; and

FIG. 10 is block diagram showing a software architecture within whichthe present disclosure may be implemented, in accordance with examples.

DETAILED DESCRIPTION

The present disclosure describes GUIs for use with eyewear devices. Inone example, the GUI is implemented as a cross (up/down/left/right) withone or more contextual menus that are always available when the userneeds them, that contain clear call-to-action menu items, and thatprovide visual feedback when an action is taken. The contextual menuitems may be designed to be memorizable in order to allow a user to takefast action.

The description that follows includes systems, methods, techniques,instruction sequences, and computing machine program products forimplementing a graphical user interface (GUI) illustrative of examplesof the disclosure. In the following description, for the purposes ofexplanation, numerous specific details are set forth in order to providean understanding of various examples of the disclosed subject matter. Itwill be evident, however, to those skilled in the art, that examples ofthe disclosed subject matter may be practiced without these specificdetails. In general, well-known instruction instances, protocols,structures, and techniques are not necessarily shown in detail.

FIG. 1A is a right side view of an example hardware configuration of aneyewear device 100, which includes a touch sensitive input surface 181.Eyewear device 100 includes a right optical assembly 180B with an imagedisplay to present images. As shown in FIG. 1A, the eyewear device 100includes the right visible light camera 114B. Eyewear device 100 caninclude multiple visible light cameras 114A-B that form a passive typeof depth-capturing camera, such as a stereo camera, of which the rightvisible light camera 114B is located on a right side (e.g., in a rightchunk 110B). The eyewear device 100 can also include a left visiblelight camera 114A on a left side (e.g., in a left chunk 110A). Thedepth-capturing camera(s) can be an active type of depth-capturingcamera that includes a single visible light camera 114B and a depthsensor (e.g., an infrared camera and an infrared emitter).

Left and right visible light cameras 114A-B are sensitive to the visiblelight range wavelength. Each of the visible light cameras 114A-B have adifferent frontward facing field of view which are overlapping to allowthree-dimensional depth images to be generated, for example, rightvisible light camera 114B has the depicted right field of view 111B.Generally, a “field of view” is the part of the scene that is visiblethrough the camera at a particular position and orientation in space.Objects or object features outside the field of view 111A-B when theimage is captured by the visible light camera are not recorded in a rawimage (e.g., photograph or picture). The field of view describes anangle range or extent in which the image sensor of the visible lightcamera 114A-B picks up electromagnetic radiation of a given scene in acaptured image of the given scene. Field of view can be expressed as theangular size of the view cone, i.e., an angle of view. The angle of viewcan be measured horizontally, vertically, or diagonally.

In an example, visible light cameras 114A-B have a field of view with anangle of view between 15° to 30° , for example 24° , and have aresolution of 480×480 pixels. The “angle of coverage” describes theangle range that a lens of visible light cameras 114A-B or infraredcamera can effectively image. Typically, the image circle produced by acamera lens is large enough to cover the film or sensor completely,possibly including some vignetting (e.g., a reduction of an image'sbrightness or saturation toward the periphery compared to the imagecenter). If the angle of coverage of the camera lens does not fill thesensor, the image circle will be visible, typically with strongvignetting toward the edge, and the effective angle of view will belimited to the angle of coverage.

Examples of such visible lights camera 114A-B include a high-resolutioncomplementary metal—oxide—semiconductor (CMOS) image sensor and a videographic array (VGA) camera, such as 640 p (e.g., 640×480 pixels for atotal of 0.3 megapixels), 720 p, or 1080 p. As used herein, the term“overlapping” when referring to field of view means the matrix of pixelsin the generated raw image(s) or infrared image of a scene overlap by30% or more. As used herein, the term “substantially overlapping” whenreferring to field of view means the matrix of pixels in the generatedraw image(s) or infrared image of a scene overlap by 50% or more.

Image sensor data from the visible light cameras 114A-B are capturedalong with geolocation data, digitized by an image processor, and storedin a memory. The captured left and right raw images captured byrespective visible light cameras 114A-B are in the two-dimensional spacedomain and comprise a matrix of pixels on a two-dimensional coordinatesystem that includes an X axis for horizontal position and a Y axis forvertical position. Each pixel includes a color attribute (e.g., a redpixel light value, a green pixel light value, and a blue pixel lightvalue); and a position attribute (e.g., an X location coordinate and a Ylocation coordinate).

To provide stereoscopic vision, visible light cameras 114A-B may becoupled to an image processor (element 312 of FIG. 3 ) for digitalprocessing along with a timestamp in which the image of the scene iscaptured. Image processor 312 includes circuitry to receive signals fromthe visible light cameras 114A-B and process those signals from thevisible light camera 114 into a format suitable for storage in thememory. The timestamp can be added by the image processor or otherprocessor, which controls operation of the visible light cameras 114A-B.Visible light cameras 114A-B allow the depth-capturing camera tosimulate human binocular vision. Depth-capturing camera provides theability to reproduce three-dimensional images based on two capturedimages from the visible light cameras 114A-B having the same timestamp.Such three-dimensional images allow for an immersive life-likeexperience, e.g., for virtual reality or video gaming.

For stereoscopic vision, a pair of raw red, green, and blue (RGB) imagesare captured of a scene at a given moment in time—one image for each ofthe left and right visible light cameras 114A-B. When the pair ofcaptured raw images from the frontward facing left and right field ofviews 111A-B of the left and right visible light cameras 114A-B areprocessed (e.g., by the image processor 312 of FIG. 3A), depth imagesare generated, and the generated depth images can be perceived by a useron the optical assembly 180A-B or other image display(s) (e.g., of amobile device). The generated depth images are in the three-dimensionalspace domain and can comprise a matrix of vertices on athree-dimensional location coordinate system that includes an X axis forhorizontal position (e.g., length), a Y axis for vertical position(e.g., height), and a Z axis for depth (e.g., distance). Each vertexincludes a color attribute (e.g., a red pixel light value, a green pixellight value, and a blue pixel light value); a position attribute (e.g.,an X location coordinate, a Y location coordinate, and a Z locationcoordinate); a texture attribute, a reflectance attribute, or anycombination thereof. The texture attribute quantifies the perceivedtexture of the depth image, such as the spatial arrangement of color orintensities in a region of vertices of the depth image.

FIGS. 1B-C are rear views of example hardware configurations of theeyewear device 100, including two different types of image displays.Eyewear device 100 is in a form configured for wearing by a user, whichare eyeglasses in the example. The eyewear device 100 can take otherforms and may incorporate other types of frameworks, for example, aheadgear, a headset, or a helmet.

In the eyeglasses example, eyewear device 100 includes a frame 105including a left rim 107A connected to a right rim 107B via a bridge 106adapted for a nose of the user. The left and right rims 107A-B includerespective apertures 175A-B which hold a respective optical element180A-B, such as a lens and a display device. As used herein, the termlens is meant to cover transparent or translucent pieces of glass orplastic having curved or flat surfaces that cause light toconverge/diverge or that cause little or no convergence or divergence.

Although shown as having two optical elements 180A-B, the eyewear device100 can include other arrangements, such as a single optical elementdepending on the application or intended user of the eyewear device 100.As further shown, eyewear device 100 includes a left chunk 110A adjacentthe left lateral side 170A of the frame 105 and a right chunk 110Badjacent the right lateral side 170B of the frame 105. The chunks 110A-Bmay be integrated into the frame 105 on the respective lateral sides170A-B (as illustrated) or implemented as separate components attachedto the frame 105 on the respective sides 170A-B. Alternatively, thechunks 110A-B may be integrated into temples (not shown) attached to theframe 105.

As shown in FIG. 1B, the optical assembly 180A-B includes a suitabledisplay matrix 170, e.g., a liquid crystal display (LCD), an organiclight-emitting diode (OLED) display, or other such display. The opticalassembly 180A-B also includes an optical layer or layers 176, which caninclude lenses, optical coatings, prisms, mirrors, waveguides, opticalstrips, and other optical components in any combination. The opticallayers 176A-N can include a prism having a suitable size andconfiguration and including a first surface for receiving light fromdisplay matrix and a second surface for emitting light to the eye of theuser.

The prism of the optical layers 176A-N extends over all or at least aportion of the respective apertures 175A-B formed in the left and rightrims 107A-B to permit the user to see the second surface of the prismwhen the eye of the user is viewing through the corresponding left andright rims 107A-B. The first surface of the prism of the optical layers176A-N faces upwardly from the frame 105 and the display matrix overliesthe prism so that photons and light emitted by the display matriximpinge the first surface. The prism is sized and shaped so that thelight is refracted within the prism and is directed towards the eye ofthe user by the second surface of the prism of the optical layers176A-N. In this regard, the second surface of the prism of the opticallayers 176A-N can be convex to direct the light towards the center ofthe eye. The prism can optionally be sized and shaped to magnify theimage projected by the display matrix 170, and the light travels throughthe prism so that the image viewed from the second surface is larger inone or more dimensions than the image emitted from the display matrix170.

In another example, the image display device of optical assembly 180A-Bincludes a projection image display as shown in FIG. 1C. The opticalassembly 180A-B includes a laser projector 150, which is a three-colorlaser projector using a scanning mirror or galvanometer. Duringoperation, an optical source such as a laser projector 150 is disposedin or on one of the temples 125A-B of the eyewear device 100. Opticalassembly 180A-B includes one or more optical strips 155A-N spaced apartacross the width of the lens of the optical assembly 180A-B or across adepth of the lens between the front surface and the rear surface of thelens.

As the photons projected by the laser projector 150 travel across thelens of the optical assembly 180A-B, the photons encounter the opticalstrips 155A-N. When a particular photon encounters a particular opticalstrip, the photon is either redirected towards the user's eye, or itpasses to the next optical strip. A combination of modulation of thelaser projector 150, and modulation of the optical strips, controlspecific photons or beams of light. In an example, a processor controlsoptical strips 155A-N by initiating mechanical, acoustic, orelectromagnetic signals. Although shown as having two optical assemblies180A-B, the eyewear device 100 can include other arrangements, such as asingle or three optical assemblies, or the optical assembly 180A-B mayhave different arrangements depending on the application or intendeduser of the eyewear device 100.

In one example, the image display includes a first (left) image displayand a second (right) image display. Eyewear device 100 includes firstand second apertures 175A-B which hold a respective first and secondoptical assembly 180A-B. The first optical assembly 180A includes thefirst image display (e.g., a display matrix 170A of FIG. 1B; or opticalstrips 155A-N′ and a projector 150A of FIG. 1C). The second opticalassembly 180B includes the second image display e.g., a display matrix170B of FIG. 1B; or optical strips 155A-N″ and a projector 150B of FIG.1C).

FIG. 1D depicts a schematic view of the eyewear device 100 including aninput surface 181 on the right hand side for receiving finger taps andgestures from at least one finger contact 179 inputted from a user. Asimilar input surface may be present on the left hand side. The inputsurface 181 is configured to detect finger touch/taps and gestures(e.g., motion) for use with a GUI displayed by the eyewear display tonavigate through and select menu options in an intuitive manner, whichenhances and simplifies the user experience.

Detection of finger taps/gestures via the input surface(s) 181 canenable several functions. For example, touching anywhere on the inputsurface 181 may cause the GUI to display or highlight an item on thescreen of the image display of the optical assembly 180A-B. Doubletapping on the input surface 181 may select an item. Sliding (e.g.,swiping) 186 a finger in one direction, e.g., from front to back, backto front, up to down, or down to up may slide or scroll in onedirection, for example, to move to a previous video, image, page, orslide. Sliding the finger in another the other may slide or scroll inthe opposite direction, for example, to move to a previous video, image,page, or slide. The input surface(s) 181 can be virtually anywhere onthe eyewear device 100.

In one example, when the identified finger gesture is a single tap onthe input surface 181, this initiates selection or pressing of agraphical user interface element in the image presented on the imagedisplay of the optical assembly 180A-B. An adjustment to the imagepresented on the image display of the optical assembly 180A-B based onthe identified finger gesture can be a primary action which selects orsubmits the graphical user interface element on the image display of theoptical assembly 180A-B for further display or execution.

Eyewear device 100 may include wireless network transceivers, forexample cellular or local area network transceivers (e.g., WiFi orBluetooth™), and run applications (also known as apps). Some of theapplications may include a web browser to navigate the Internet, anapplication to place phone calls, video or image codecs to watch videosor interact with pictures, codecs to listen to music, a turn-by-turnnavigation application (e.g., to enter in a destination address and viewmaps), an augmented reality application, and an email application (e.g.,to read and compose emails). Gestures inputted on the input surface 181can be used to manipulate and interact with the displayed content on theimage display and control the applications.

Execution of the programming (element 345 of FIG. 3 ) by the processor(element 332 of FIG. 3 ) further configures the eyewear device 100 toidentify a finger gesture on the input surface 181 of the eyewear device100 by detecting at least one detected touch event based on variation ofthe tracked movement of the eyewear device 100 over a time period.Execution of the programming (element 345 of FIG. 3 ) by the processor(element 332 of FIG. 3 ) further configures the eyewear device 100 toadjust the image presented on the image display of the optical assembly180A-B of the eyewear device 100 based on the identified finger gesture.

FIG. 1E is a top cross-sectional view of a left chunk 110A of theeyewear device 100 of FIGS. 1B and 1C depicting a circuit board 140A. Inthe example, the left chunk 110A includes a flexible printed circuitboard 140A. The left chunk 110A may be integrated into or connected tothe frame 105 on the left lateral side 170A within housing 211. In someexamples, the right chunk 110B includes one or more components similarto component(s) in the left chunk 110A.

FIGS. 2A, 2B, and 2C depict a schematic view of operation illustrating apress (FIG. 2A), a hold (FIG. 2B), and a release (FIG. 2C) performed onan input surface 181 of the eyewear device 100. FIG. 2A illustrates thefinger 202 of a user's hand 204 contacting the input surface 181 in afinger contact area 210A. FIG. 2B illustrates identification of thefinger 202 in a more precise finger contact area 210B on the inputsurface 181. FIG. 2C illustrates a release of the finger 202 from theinput surface 181 from the finger contact area 210C.

FIGS. 2D and 2E illustrate a finger 202 contacting an input surface 181and swiping right, left, up, and down. In FIG. 2D, the finger 202contacts the input surface 181 at an initial point of contact 212A.Sliding the finger 202 to the right/forward toward a right/forward pointof contact 212B is associated with a first action and sliding the fingerto the left/backward toward a left/backward point of contact 212C isassociated with a second action. In FIG. 2E, the finger 202 contacts theinput surface 181 at the initial point of contact 212A. Sliding thefinger 202 up toward an upper point of contact 212D is associated with athird action and sliding the finger down toward a lower point of contact212E is associated with a fourth action. Additionally, tapping the inputsurface 181 is associated with a fifth action.

FIGS. 2F and 2G illustrates a GUI menu template 250. The template 250includes a swipe forward label 252, a swipe backward label 254, a taplabel 256, and a swipe down label 258. Additionally, the template mayinclude a label associated with a swipe up. Each label may be associatedwith a word or symbol corresponding to a menu action (e.g., send, store,etc.). The template may also include directional arrows 262A-Cindicating finger movement and a bullseye 260 indicating a tap needed toperform a particular menu action.

FIG. 2F illustrates a user finger position 270 on the input surface 181.While the user's finger is within the bullseye region 260, all actionsavailable through the GUI are available for selection as indicated byall labels and arrows being in a uniform font, color, or brightness. Asthe user's finger is slid out of the bullseye region 260 toward one ofthe available directions, the action for selection (and itscorresponding arrow) is emphasized (e.g., it is emphasized or the otheractions are deemphasized/greyed out/removed) with respect to the otheractions/arrows (e.g., swiping right/forward in FIG. 2G to fingerposition 272). Releasing the finger 202 from the input surface 210 whenthe action for selection is emphasized results in selection of thataction. Moving the finger 202 back toward the bullseye region 260without release will not result in a selection and all actions willagain be available for selection.

FIG. 2H depicts a remote accessory 280 embodied as a ring. The remoteaccessory includes an input device such as an input surface similar toinput surface 181 or a joystick 282 as illustrated. A user may actuatethe joystick to perform operations associated with the GUI. Right andleft movement of the joystick 282 is associated with forward andbackward actions on the input surface 181, downward movement isassociated with a down action, and depressing the joystick 282 isassociated with a tap. A selection may be made using the remoteaccessory 280 through a quick press or by moving the joystick in thedirection 284 associated with the desired action followed by a quickrelease. The GUI and associated actions available when the remoteaccessory is in use may be different that the GUI and associated actionsavailable when using the input surface 181.

FIG. 3 depicts and eyewear system 300 for implementing GUIs as describedherein in an eyewear device 100. The system includes the eyewear device100, a mobile device 390, and a host computer, such as a server system398. The host computer may be a personal computer, embedded high speedprocessor, GPU, FPGA or other processing system. Server system 398 maybe one or more computing devices as part of a service or networkcomputing system, for example, that include a processor, a memory, andnetwork communication interface to communicate over a network 395 (e.g.,the Internet) with the mobile device 390 and eyewear device 100. It isnot required for the server system 398 to communicate with the mobiledevice 390 or the eyewear device 100.

Finger tap and gesture detection programming 345, GUI selectionprogramming 347, and GUI display programming 349 are stored forexecution by eyewear device 100. Although shown in the memory 334 of thehigh-speed circuitry 330, one or more of the programming instructionscan be stored and executed in the low power processor 322.

The eyewear system additionally includes an input tracker 310 receivinginput from the input surface 181 to identify finger taps and gestures, amobile device 390, and a server system 398 connected via variousnetworks. Eyewear device 100 is connected with a host computer. Forexample, the eyewear device 100 is paired with the mobile device 390 viathe high-speed wireless connection 337 or connected to the server system398 via the network 395.

The mobile device 390 may be a smartphone, tablet, laptop computer,access point, or any other such device capable of connecting witheyewear device 100 using both a low-power wireless connection 325 and ahigh-speed wireless connection 337. Mobile device 390 is connected toserver system 398 and network 395. The network 395 may include anycombination of wired and wireless connections.

Eyewear device 100 includes the input tracker 310 including inertiameasuring unit 319 and a depth-capturing camera, such as at least one ofthe visible light cameras 114A-B. Eyewear device 100 further includestwo image displays of the optical assembly 180A-B (one associated withthe left lateral side 170A and one associated with the right lateralside 170B). Eyewear device 100 also includes image display driver 342,image processor 312, low-power circuitry 320, and high-speed circuitry330. Image display of optical assembly 180A-B is for presenting imagesand videos, which can include a sequence of images. Image display driver342 is coupled to the image display of optical assembly 180A-B tocontrol the image display of optical assembly 180A-B to present theimages. The components shown in FIG. 3 for the eyewear device 100 arelocated on one or more circuit boards, for example a PCB or flexiblePCB, in the rims 107A-B or temples 125A-B. Alternatively, oradditionally, the depicted components can be located in the chunks110A-B, frame 105, hinges 126A-B, or bridge 106 of the eyewear device100.

As shown, memory 334 further includes tap and gesture detectionprogramming 345 to perform a subset or all of the functions describedherein for gesture detection, including GUI selection programming 347 toperform a subset or all of the functions described herein for selectingGUIs, and GUI display programming 349 to perform a subset or all of thefunctions described herein for displaying GUIs.

High-speed circuitry 330 includes high-speed processor 332, memory 334,and high-speed wireless circuitry 336. In the example, the image displaydriver 342 is coupled to the high-speed circuitry 330 and is operated bythe high-speed processor 332 in order to drive the left and right imagedisplays of the optical assembly 180A-B. High-speed processor 332 may beany processor capable of managing high-speed communications andoperation of any general computing system needed for eyewear device 100.High-speed processor 332 includes processing resources needed formanaging high-speed data transfers on high-speed wireless connection 337to a wireless local area network (WLAN) using high-speed wirelesscircuitry 336.

The high-speed processor 332 may execute firmware that includes the tapand gesture detection programming 345, including GUI selectionprogramming 347, GUI display programming 349, and an operating system,such as a LINUX operating system or other such operating system of theeyewear device 100 and the operating system is stored in memory 334 forexecution. In addition to any other responsibilities, the high-speedprocessor 332 executing a software architecture for the eyewear device100 manages data transfers with high-speed wireless circuitry 336. Insome examples, high-speed wireless circuitry 336 implements Institute ofElectrical and Electronic Engineers (IEEE) 802.11 communicationstandards, also referred to herein as Wi-Fi. In other examples, thehigh-speed wireless circuitry 336 implements other high-speedcommunication standards.

Low-power wireless circuitry 324 and the high-speed wireless circuitry336 of the eyewear device 100 can include short range transceivers(Bluetooth™) and wireless wide, local, or wide area network transceivers(e.g., cellular or WiFi). Mobile device 390, including the transceiverscommunicating via the low-power wireless connection 325 and high-speedwireless connection 337, may be implemented using details of thearchitecture of the eyewear device 100, as can other elements of network395.

Memory 334 includes any storage device capable of storing various dataand applications, including, among other things, programminginstructions, camera data generated by the left and right visible lightcameras 114A-B, and the image processor 312, as well as images andvideos generated for display by the image display driver 342 on theimage displays of the optical assembly 180A-B. While memory 334 is shownas integrated with high-speed circuitry 330, in other examples, memory334 may be an independent standalone element of the eyewear device 100.In certain such examples, electrical routing lines may provide aconnection through a chip that includes the high-speed processor 332from the image processor 312 or low-power processor 322 to the memory334. In other examples, the high-speed processor 332 may manageaddressing of memory 334 such that the low-power processor 322 will bootthe high-speed processor 332 any time that a read or write operationinvolving memory 334 is needed.

The processor 332 of the eyewear device 100 is coupled to the inputtracker 310, the depth-capturing camera (visible light cameras 114A-B;or visible light camera 114A and a depth sensor, which is not shown),the image display driver 342, and the memory 334. Eyewear device 100 canperform all or a subset of any of the following functions describedbelow as a result of the execution of the gesture detection programming345, GUI selection programming 347, and GUI display programming 349 inthe memory 334 by the processor 332 of the eyewear device 100.

Execution of the tap and gesture detection programming 345, GUIselection programming 347, and GUI display programming 349 by theprocessor 332 configures the eyewear device 100 to perform functions,including functions to track, via the input tracker 310, finger movementon the input surface 181 of the eyewear device 100 from the at least onefinger contact 179 input by the user on the input surface 181. Eyeweardevice 100 identifies a finger gesture on the input surface 181 of theeyewear device by detecting at least one detected touch event based onvariation of the tracked movement. Eyewear device 100 adjusts the GUIpresented on the image display 180A-B of the eyewear device 100 based onthe identified finger gestures and taps.

Output components of the eyewear device 100 include visual components,such as the left and right image displays of optical assembly 180A-B asdescribed in FIGS. 1B-C (e.g., a display such as a liquid crystaldisplay (LCD), a plasma display panel (PDP), a light emitting diode(LED) display, a projector, or a waveguide). Left and right imagedisplays of optical assembly 180A-B can present images, such as in avideo. The image displays of the optical assembly 180A-B are driven bythe image display driver 342.

The output components of the eyewear device 100 further include acousticcomponents (e.g., speakers), haptic components (e.g., a vibratorymotor), other signal generators, and so forth. The input components ofthe eyewear device 100, the mobile device 390, and server system 398,may include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, a photo-opticalkeyboard, or other alphanumeric input components), point-based inputcomponents (e.g., a mouse, a touchpad, a trackball, a joystick, a motionsensor, or other pointing instruments), tactile input components (e.g.,a physical button, a touch screen that provides location and force oftouches or touch gestures, or other tactile input components), audioinput components (e.g., a microphone), and the like.

Eyewear device 100 may optionally include additional peripheral deviceelements. Such peripheral device elements may include biometric sensors,additional sensors, or display elements integrated with eyewear device100. Peripheral device elements may include any I/O components includingoutput components, motion components, position components, or any othersuch elements described herein.

FIG. 4A is an example GUI 400 presented on the display of the eyeweardevice 100 when a finger contacts an input surface on the left side ofthe eyewear device 100. The GUI 400 has a left hand perspective; anddepicts a virtual pointer 402A (e.g., an image of a left hand with anindex finger extended) and a virtual input surface 404A. The virtualpointer 402A corresponds to the finger on the input surface on the leftside of the eyewear device 100, the virtual input surface 404Acorresponds to the input surface on the left side of the eyewear device100, and the relationship between the positional relationship of thevirtual pointer 402A on the virtual input surface 404A corresponds theposition of the finger on the input surface on the left side of theeyewear device 100. Although depicted as a 2D image, the GUI 400 may bepresented on the display of the eyewear device 100 as a 3D image havingdepth perceived by the user wearing the eyewear device.

The available actions in the GUI 400 include an associated label havinga symbol and text. Swiping up on the input surface on the left side andreleasing (associated with an eye symbol and the text “View”) results inan action to view an image on the eyewear device 100. Swiping down onthe input surface on the left side and releasing (associated with an Xsymbol and the text “Dismiss”) results in no action. Swipingforward/left on the input surface on the left side and releasing(associated with a checkmark in a circle symbol and the text “Send toRecent”) results in an action to send an image to the same recipientthat an immediately prior image was sent. Swiping backward/right on theinput surface on the left side and releasing (associated with a squarewith a plus sign and the text “Add to My Story”) results in an action tosave an image to a particular area of a social media application.

FIG. 4B depicts a template for setting up a left hand perspective GUIsuch as GUI 400. The template includes an orientation sphere 420.Additionally, the template includes a label 422A-E for each availabletap/gesture and a corresponding directional arrow 424A-E.

FIG. 4C is an example GUI 450 presented on the display of the eyeweardevice 100 when a finger contacts an input surface 181 on the right sideof the eyewear device 100. The GUI 450 has a right hand perspective; anddepicts a virtual pointer 402B (e.g., an image of a right hand with anindex finger extended) and a virtual input surface 404B. The virtualpointer 402B corresponds to the finger on the input surface 181 on theright side of the eyewear device 100, the virtual input surface 404Bcorresponds to the input surface on the right side of the eyewear device100, and the relationship between the positional relationship of thevirtual pointer 402B on the virtual input surface 404B corresponds theposition of the finger on the input surface on the right side of theeyewear device 100. The available actions in the GUI 450 are the same asin the GUI 400 from a right hand perspective (e.g., swipingforward/right on the input surface 181 rather than forward/left resultsin an action to send an image to the same recipient that an immediatelyprior image was sent). Although depicted as having the same associatedactions, the left hand GUI 400 and the right had GUI 450 may beassociated with different actions.

FIG. 4D depicts a template for setting up a right hand perspective GUIsuch as GUI 450. The template includes an orientation sphere 420.Additionally, the template includes a label 422A-E for each availabletap/gesture and a corresponding directional arrow 424A-E.

FIGS. 4A-D illustrate one benefit of some of the example GUIs describedherein. With the input surface 181 on the side of the eyewear, it is notintuitive in conventional GUIs designed for 2D systems what a swipeforward or a swipe backward will select. By displaying a virtual inputsurface 404 in 3D along with labels relatively positioned with respectto the virtual input surface and having left/right hand perspectiveviews depending on which side of the eyewear device the user isinteracting, an intuitive display is provided for quickly and easilyselecting desired actions.

FIG. 4E depicts another GUI 470 for selecting from among multipleoptions. The GUI 470 includes a virtual pointer 402A and a selectionring 472. By performing a downward swipe on the input interface, a usercan cause the elements of the ring (e.g., a user's friends) to rotate ina counterclockwise direction. By performing an upward swipe on the inputinterface, a user can cause the elements of the ring (e.g., a user'sfriends) to rotate in a clockwise direction. The GUI includes aselection box 474. A user can select a particular element from the ring472 by positioning that element within the selection box 474 and tappingthe input surface 181.

FIG. 4F is another GUI 480. The GUI 480 has a straight on perspective(as opposed to a right hand or left hand perspective), e.g., for usewith a remote accessory such as a depicted and described with respect toFIG. 2H. The available actions in the GUI 480 are the same as in theGUIs 400 and 450, but with a straight on perspective (e.g., swipingright on the input surface 181 rather than forward/right or forward leftresults in an action to send an image to the same recipient that animmediately prior image was sent). Although depicted as having the sameassociated actions, the GUIs 400, 450 and 480 may each be associatedwith different actions. FIG. 4G is another GUI 490 similar to the GUI470 from a straight on perspective. The ring 472 in GUI 490 is circular(rather than oval as in FIG. 4E) to facilitate user perception/selectionwhen a device such as a remote accessory that benefits from a straighton perspective is used to interface with the eyewear device 100.

FIGS. 5A, 5B, and 5C are illustrations of a GUI interaction for imagesreceived from another user. In FIG. 5A, a notification 500 displays onthe image displays 180A, B of the eyewear device 100 when an image/videomessage is received by the eyewear device 100 (or an associated mobiledevice). In FIG. 5B, a GUI 502 with available actions for interacting(e.g, “View Snap” on tap, “Dismiss Notification” on swipe down) displayson the image displays 180A, B of the eyewear device 100. In FIG. 5C,when the user swipes down, the action is confirmed by highlighting theaction label taken or deemphasizing the action not being taken.

FIGS. 5D, 5E, and 5F are illustrations of a GUI interaction forreceiving and responding to messages from another user. In FIG. 5D, anotification 504 displays on image displays 180A, B when a text messageis received by the eyewear device 100 (or an associated mobile device).Additionally, a GUI 506 with available actions for interacting (e.g.,“Answer with Voice-to-Chat” on swipe forward, “Quick Answers” selectionon swipe backward, “View Chat History” on tap, and “Dismiss” on swipedown) displays. In FIG. 5E, when the user swipes forward, the action isconfirmed by highlighting the action label taken and the actions notbeing taken are deemphasized. In FIG. 5F, a new GUI 508 displays onimage displays 180A, B for use with the voice-to-chat answer selectionand a response window 510 is opened.

FIGS. 5G, 5H, and 5I are illustrations of a GUI interaction for sendingan image/video to another user. In FIG. 5G, a notification 512 displayson image displays 180A, B when an image/video is captured by the eyeweardevice 100 (or an associated mobile device). Additionally, a GUI 514with available actions for interacting (e.g, “Send to Ilter” on swipeforward, “Select Recipients” selection on swipe backward, “Open CreativeTools” on tap, and “Dismiss” on swipe down) displays. In FIG. 5H, whenthe user swipes backward, the action is confirmed by highlighting theaction label taken and the actions not being taken are deemphasized. InFIG. 5I, a new GUI 516 displays on image displays 180A, B for use withthe recipient selection.

FIG. 6 is a block diagram of a GUI engine 600 for implementing a GUI onthe eyewear device 100. The GUI engine includes a touch module 602, aGUI selection module 604, and a GUI display module 606. The touch module602 is a computer module for execution on a processor of a computerplatform that receives input from input surface 181 and identifiedfinger taps, touches, holds, slides, and swipes in response to thereceived input.

The GUI selection module 604 is a computer module for execution on aprocessor of a computer platform that identifies which GUI to display tothe user. The GUI selection module 604 monitors applications running onthe computer platform and selects a GUI based on notifications from theapplications. For example, if a text application receives an incomingemail and provides a notification, the GUI selection module 604 selectsone GUI. If another application such as a camera application provides anotification that a picture was taken, the GUI selection module 604selects another GUI. Additionally, the GUI selection module selects aGUI based on which input surface of the eyewear is being used (e.g., aleft hand perspective GUI when the left input surface is pressed, aright hand perspective GUI when the right input surface is pressed, anda straight on perspective when a remote accessory is pressed.

The GUI display module 606 is a computer module for execution on aprocessor of a computer platform that displays the GUIs to the user onan optical assembly 180 of the eyewear device 100. The GUI displaymodule 606 displays the selected GUIs on the image displays 180A, B.Additionally, the GUI display module transitions from one GUI toanother. For example, if the user presses the left input surface andthen the right input surface, the GUI display module 606 will transitionfrom the left hand perspective GUI 400 to the right hand perspective GUI450.

FIGS. 7A, 7B, and 7C depict flow charts for implementing a graphicaluser interface using the GUI engine 600. Although the steps aredescribed with reference to the eyewear 100 described herein, othereyewear for implementing the steps of FIGS. 7A, 7B, and 7C will beunderstood by one of skill in the art from the description herein.Additionally, it is contemplated that one or more of the steps of FIGS.7A, 7B, or 7C may be omitted, performed simultaneously or in series,additional steps may be added, and steps may be performed in an orderother than illustrated and described.

Flow chart 700 depicts steps for selecting an action using a GUI. Atstep 702, monitor the application(s). The GUI selection module 604 ofthe eyewear device 100 (e.g., processor(s) 322/332 and associatedprogramming instructions implementing the GUI selection module 604)monitors applications running on the eyewear device (e.g., a textingapp, a camera app, etc.) or an associated mobile device. At step 704,determine the state of the application(s). The GUI selection module 604of the eyewear device 100 monitors applications for notification events(e.g., a message received by a texting app, an image captured by acamera app, etc.).

At step 706, select a GUI for display. The GUI selection module 604 ofthe mobile device selects a GUI. The GUI selection is based on the inputsurface 181, the state of the application(s), or a combination thereof.For example, a processor implementing the GUI selection module selects aleft hand perspective GUI 400 when the input surface on the left handside of the eyewear is in use, a right hand perspective GUI 450 when theinput surface 181 on the right hand side of the eyewear is in use, and astraight on perspective GUI 480 when the input is from an accessorydevice. Additionally, different actions are available through the GUIwhen a text message is received by a texting app and when an image iscaptured by a camera app.

At step 708, detect a finger touch on the input surface/touch pad. Thetouch module 602 detects the finger touch based on input from the inputsurface 181. At step 710, present the GUI. The GUI display module 606displays the GUI such that it becomes visible on the optical assembly180 of the eyewear 100 when the finger is detected and remains visibleuntil it is removed.

At step 712, identify a finger gesture on the input surface. The touchmodule 602 monitors the position of the finger while it is on the inputsurface and processes the positions to identify gestures. For example, amovement in a generally downward direction is perceived as a swipe down.

At step 714, adjust the GUI. The GUI display module 606 adjusts the GUIin response to the finger movement on the input surface. The processorimplementing the GUI display module 606 may display the actual positionof the finger on the input surface in a corresponding virtual positionon a virtual input surface on the GUI. Additionally, as the finger ismoved in a particular direction, the label associated with thatdirection may be highlighted to indicate that action will be selectedupon removal of the finger from the input surface. At step 716, detect afinger release and, at step 718, perform the action. The touch module602 detects the finger release based on input from the input surface andthe eyewear performs the selected action associated with the identifiedgesture.

Flow chart 730 (FIG. 7B) depicts steps for performing actions inresponse to a tap or a touch. At step 732, detect a finger touch. Thetouch module 602 detects the finger touch based on input from the inputsurface. At step 734, monitor the time to release the finger. The touchmodule 602 starts a timer when the finger touch is detected, which runsuntil the finger is removed. At decision block 736, compare themonitored time to a predetermined time (e.g., 100 ms). If the time isless than the predetermined time, perform the default action in step 738(e.g., a default action option from the GUI or another predefinedaction). Otherwise, display the GUI without performing the defaultaction in step 740.

Flow chart 760 (FIG. 7C) depicts steps for transitioning from a firstGUI to a second GUI. At step 762 detect a second finger touch. The touchmodule 602 detects the second finger touch based on input from the inputsurface. Assuming a first GUI (e.g., GUI 400) is currently beingdisplayed, at step 764, transition to a second GUI (e.g., GUI 450). Atstep 766, detect a remote accessory controller. The touch module 602 ora separate remote detection module (not shown) detects the remoteaccessory controller based on a transmission from the remote accessorycontroller that includes control instructions. Assuming the first or thesecond GUI are currently being displayed, at step 768, transition to a3rd GUI (e.g., GUI 480) associated with the accessory controller.

FIG. 8 is a high-level functional block diagram of an example clientdevice including a mobile device 390 that communicates via network 395with server system 398 of FIG. 3 . Shown are elements of a touch screentype mobile device 390, although other non-touch type mobile devicessuch as eyewear device 100 can be used. Examples of touch screen typemobile devices that may be used include (but are not limited to) a smartphone, a personal digital assistant (PDA), a tablet computer, a laptopcomputer, or other portable device. However, the structure and operationof the touch screen type devices is provided by way of example, and thesubject technology as described herein is not intended to be limitedthereto. For purposes of this discussion, FIG. 8 therefore provides ablock diagram illustration of an example mobile device 390 having anoptional touch screen display 890 for displaying content and receivinguser input as (or as part of) the user interface. Mobile device 390 alsoincludes a camera(s) 870, such as visible light camera(s), and amicrophone 880.

The activities that are the focus of discussions here involve creatingand interfacing with graphical user interfaces. The graphical userinterfaces may be implemented using a graphical user interface 600stored in memory 840A for execution by CPU 830.

As shown in FIG. 8 , the mobile device 390 includes at least one digitaltransceiver (XCVR) 810, shown as WWAN XCVRs, for digital wirelesscommunications via a wide area wireless mobile communication network395. The mobile device 390 also includes additional digital or analogtransceivers, such as short range XCVRs 820 for short-range networkcommunication, such as via NFC, VLC, DECT, ZigBee, Bluetooth™, or WiFi.For example, short range XCVRs 820 may take the form of any availabletwo-way wireless local area network (WLAN) transceiver of a type that iscompatible with one or more standard protocols of communicationimplemented in wireless local area networks, such as one of the Wi-Fistandards under IEEE 802.11 and 4G LTE.

To generate location coordinates for positioning of the mobile device390, the mobile device 390 can include a global positioning system (GPS)receiver (not shown). Alternatively, or additionally, the mobile device390 can utilize either or both the short range XCVRs 820 and WWAN XCVRs810 for generating location coordinates for positioning. For example,cellular network, WiFi, or Bluetooth™ based positioning systems cangenerate very accurate location coordinates, particularly when used incombination. Such location coordinates can be transmitted to the eyeweardevice over one or more network connections via XCVRs 820.

The transceivers 810, 820 (network communication interface) conforms toone or more of the various digital wireless communication standardsutilized by modern mobile networks. Examples of WWAN transceivers 810include (but are not limited to) transceivers configured to operate inaccordance with Code Division Multiple Access (CDMA) and 3rd GenerationPartnership Project (3 GPP) network technologies including, for exampleand without limitation, 3 GPP type 2 (or 3 GPP2) and LTE, at timesreferred to as “4G.” For example, the transceivers 810, 820 providetwo-way wireless communication of information including digitized audiosignals, still image and video signals, web page information for displayas well as web related inputs, and various types of mobile messagecommunications to/from the mobile device 390 user identificationstrategies.

Several of these types of communications through the transceivers 810,820 and a network, as discussed previously, relate to protocols andprocedures in support of communications with the server system 398 forobtaining and storing friend device capabilities. Such communications,for example, may transport packet data via the short range XCVRs 820over the wireless connections of network 395 to and from the serversystem 398 as shown in FIG. 3 . Such communications, for example, mayalso transport data utilizing IP packet data transport via the WWANXCVRs 810 over the network (e.g., Internet) 395 shown in FIG. 3 . BothWWAN XCVRs 810 and short range XCVRs 820 connect through radio frequency(RF) send-and-receive amplifiers (not shown) to an associated antenna(not shown).

The mobile device 390 further includes a microprocessor 830, shown as aCPU, sometimes referred to herein as the host controller. A processor isa circuit having elements structured and arranged to perform one or moreprocessing functions, typically various data processing functions.Although discrete logic components could be used, the examples utilizecomponents forming a programmable CPU. A microprocessor for exampleincludes one or more integrated circuit (IC) chips incorporating theelectronic elements to perform the functions of the CPU. The processor830, for example, may be based on any known or available microprocessorarchitecture, such as a Reduced Instruction Set Computing (RISC) usingan ARM architecture, as commonly used today in mobile devices and otherportable electronic devices. Of course, other processor circuitry may beused to form the CPU 830 or processor hardware in smartphone, laptopcomputer, and tablet.

The microprocessor 830 serves as a programmable host controller for themobile device 390 by configuring the mobile device to perform variousoperations, for example, in accordance with instructions or programmingexecutable by processor 830. For example, such operations may includevarious general operations of the mobile device, as well as operationsrelated to performance metric monitoring, generating graphical userinterfaces, reporting to server system 398, and gating. Although aprocessor may be configured by use of hardwired logic, typicalprocessors in mobile devices are general processing circuits configuredby execution of programming.

The mobile device 390 includes a memory or storage device system, forstoring data and programming. In the example, the memory system mayinclude a flash memory 840A and a random access memory (RAM) 840B. TheRAM 840B serves as short term storage for instructions and data beinghandled by the processor 830, e.g., as a working data processing memory.The flash memory 840A typically provides longer term storage.

Hence, in the example of mobile device 390, the flash memory 840A isused to store programming or instructions for execution by the processor830. Depending on the type of device, the mobile device 390 stores andruns a mobile operating system through which specific applications run,including applications 600, 700, 730, and 760. Examples of mobileoperating systems include Google Android®, Apple iOS® (I-Phone or iPaddevices), Windows Mobile®, Amazon Fire OS®, RIM BlackBerry® operatingsystem, or the like.

FIG. 9 is a diagrammatic representation of a machine 900 within whichinstructions 908 (e.g., software, a program, an application, an applet,an app, or other executable code) for causing the machine 900 to performany one or more of the methodologies discussed herein may be executed.For example, the instructions 908 may cause the machine 900 to executeany one or more of the methods described herein. The instructions 908transform the general, non-programmed machine 900 into a particularmachine 900 programmed to carry out the described and illustratedfunctions in the manner described. The machine 900 may operate as astandalone device or may be coupled (e.g., networked) to other machines.In a networked deployment, the machine 900 may operate in the capacityof a server machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 900 may comprise, but not be limitedto, a server computer, a client computer, a personal computer (PC), atablet computer, a laptop computer, a netbook, a set-top box (STB), aPDA, an entertainment media system, a cellular telephone, a smart phone,a mobile device, a wearable device (e.g., a smart watch), a smart homedevice (e.g., a smart appliance), other smart devices, a web appliance,a network router, a network switch, a network bridge, or any machinecapable of executing the instructions 908, sequentially or otherwise,that specify actions to be taken by the machine 900. Further, while onlya single machine 900 is illustrated, the term “machine” shall also betaken to include a collection of machines that individually or jointlyexecute the instructions 908 to perform any one or more of themethodologies discussed herein.

The machine 900 may include processors 902, memory 904, and I/Ocomponents 942, which may be configured to communicate with each othervia a bus 944. In an example, the processors 902 (e.g., a CentralProcessing Unit (CPU), a Reduced Instruction Set Computing (RISC)processor, a Complex Instruction Set Computing (CISC) processor, aGraphics Processing Unit (GPU), a Digital Signal Processor (DSP), anASIC, a Radio-Frequency Integrated Circuit (RFIC), another processor, orany suitable combination thereof) may include, for example, a processor906 and a processor 910 that execute the instructions 908. The term“processor” is intended to include multi-core processors that maycomprise two or more independent processors (sometimes referred to as“cores”) that may execute instructions contemporaneously. Although FIG.9 shows multiple processors 902, the machine 900 may include a singleprocessor with a single core, a single processor with multiple cores(e.g., a multi-core processor), multiple processors with a single core,multiple processors with multiples cores, or any combination thereof

The memory 904 includes a main memory 912, a static memory 914, and astorage unit 916, both accessible to the processors 902 via the bus 944.The main memory 904, the static memory 914, and storage unit 916 storethe instructions 908 embodying any one or more of the methodologies orfunctions described herein. The instructions 908 may also reside,completely or partially, within the main memory 912, within the staticmemory 914, within machine-readable medium 918 (e.g., a non-transitorymachine-readable storage medium) within the storage unit 916, within atleast one of the processors 902 (e.g., within the processor's cachememory), or any suitable combination thereof, during execution thereofby the machine 900.

Furthermore, the machine-readable medium 918 is non-transitory (in otherwords, not having any transitory signals) in that it does not embody apropagating signal. However, labeling the machine-readable medium 918“non-transitory” should not be construed to mean that the medium isincapable of movement; the medium should be considered as beingtransportable from one physical location to another. Additionally, sincethe machine-readable medium 918 is tangible, the medium may be amachine-readable device.

The I/O components 942 may include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and so on. The specific I/Ocomponents 942 that are included in a particular machine will depend onthe type of machine. For example, portable machines such as mobilephones may include a touch input device or other such input mechanisms,while a headless server machine will likely not include such a touchinput device. It will be appreciated that the I/O components 942 mayinclude many other components that are not shown in FIG. 9 . In variousexamples, the I/O components 942 may include output components 928 andinput components 930. The output components 928 may include visualcomponents (e.g., a display such as a plasma display panel (PDP), alight emitting diode (LED) display, a liquid crystal display (LCD), aprojector, or a cathode ray tube (CRT)), acoustic components (e.g.,speakers), haptic components (e.g., a vibratory motor, resistancemechanisms), other signal generators, and so forth. The input components930 may include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, a photo-opticalkeyboard, or other alphanumeric input components), point-based inputcomponents (e.g., a mouse, a touchpad, a trackball, a joystick, a motionsensor, or another pointing instrument), tactile input components (e.g.,a physical button, a touch screen that provides location, force oftouches or touch gestures, or other tactile input components), audioinput components (e.g., a microphone), and the like.

In further examples, the I/O components 942 may include biometriccomponents 932, motion components 934, environmental components 936, orposition components 938, among a wide array of other components. Forexample, the biometric components 932 include components to detectexpressions (e.g., hand expressions, facial expressions, vocalexpressions, body gestures, or eye tracking), measure biosignals (e.g.,blood pressure, heart rate, body temperature, perspiration, or brainwaves), identify a person (e.g., voice identification, retinalidentification, facial identification, fingerprint identification, orelectroencephalogram-based identification), and the like. The motioncomponents 934 include acceleration sensor components (e.g.,accelerometer), gravitation sensor components, rotation sensorcomponents (e.g., gyroscope), and so forth. The environmental components936 include, for example, illumination sensor components (e.g.,photometer), temperature sensor components (e.g., one or morethermometers that detect ambient temperature), humidity sensorcomponents, pressure sensor components (e.g., barometer), acousticsensor components (e.g., one or more microphones that detect backgroundnoise), proximity sensor components (e.g., infrared sensors that detectnearby objects), gas sensors (e.g., gas detection sensors to detectionconcentrations of hazardous gases for safety or to measure pollutants inthe atmosphere), or other components that may provide indications,measurements, or signals corresponding to a surrounding physicalenvironment. The position components 938 include location sensorcomponents (e.g., a GPS receiver component), altitude sensor components(e.g., altimeters or barometers that detect air pressure from whichaltitude may be derived), orientation sensor components (e.g.,magnetometers), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 942 further include communication components 940operable to couple the machine 900 to network 102 and client devices 110via a coupling 924 and a coupling 926, respectively. For example, thecommunication components 940 may include a network interface componentor another suitable device to interface with the network 102. In furtherexamples, the communication components 940 may include wiredcommunication components, wireless communication components, cellularcommunication components, Near Field Communication (NFC) components,Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components,and other communication components to provide communication via othermodalities. The devices 110 may be another machine or any of a widevariety of peripheral devices (e.g., a peripheral device coupled via aUSB).

Moreover, the communication components 940 may detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 940 may include Radio Frequency Identification(RFID) tag reader components, NFC smart tag detection components,optical reader components (e.g., an optical sensor to detectone-dimensional bar codes such as Universal Product Code (UPC) bar code,multi-dimensional bar codes such as Quick Response (QR) code, Azteccode, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2Dbar code, and other optical codes), or acoustic detection components(e.g., microphones to identify tagged audio signals). In addition, avariety of information may be derived via the communication components940, such as location via Internet Protocol (IP) geolocation, locationvia Wi-Fi® signal triangulation, location via detecting an NFC beaconsignal that may indicate a particular location, and so forth.

The various memories (e.g., memory 904, main memory 912, static memory914, memory of the processors 902), storage unit 916 may store one ormore sets of instructions and data structures (e.g., software) embodyingor used by any one or more of the methodologies or functions describedherein. These instructions (e.g., the instructions 908), when executedby processors 902, cause various operations to implement the disclosedexamples.

The instructions 908 may be transmitted or received over the network102, using a transmission medium, via a network interface device (e.g.,a network interface component included in the communication components940) and using any one of a number of well-known transfer protocols(e.g., hypertext transfer protocol (HTTP)). Similarly, the instructions908 may be transmitted or received using a transmission medium via thecoupling 926 (e.g., a peer-to-peer coupling) to the devices 110.

FIG. 10 is a block diagram 1000 illustrating a software architecture1004, which can be installed on any one or more of the devices describedherein. The software architecture 1004 is supported by hardware such asa machine 1002 that includes processors 1020, memory 1026, and I/Ocomponents 1038. In this example, the software architecture 1004 can beconceptualized as a stack of layers, where each layer provides aparticular functionality. The software architecture 1004 includes layerssuch as an operating system 1012, libraries 1010, frameworks 1008, andapplications 1006. Operationally, the applications 1006 invoke API calls1050 through the software stack and receive messages 1052 in response tothe API calls 1050.

The operating system 1012 manages hardware resources and provides commonservices. The operating system 1012 includes, for example, a kernel1014, services 1016, and drivers 1022. The kernel 1014 acts as anabstraction layer between the hardware and the other software layers.For example, the kernel 1014 provides memory management, processormanagement (e.g., scheduling), component management, networking, andsecurity settings, among other functionality. The services 1016 canprovide other common services for the other software layers. The drivers1022 are responsible for controlling or interfacing with the underlyinghardware. For instance, the drivers 1022 can include display drivers,camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flashmemory drivers, serial communication drivers (e.g., Universal Serial Bus(USB) drivers), WI-FI® drivers, audio drivers, power management drivers,and so forth.

The libraries 1010 provide a low-level common infrastructure used by theapplications 1006. The libraries 1010 can include system libraries 1018(e.g., C standard library) that provide functions such as memoryallocation functions, string manipulation functions, mathematicfunctions, and the like. In addition, the libraries 1010 can include APIlibraries 1024 such as media libraries (e.g., libraries to supportpresentation and manipulation of various media formats such as MovingPicture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC),Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC),Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group(JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries(e.g., an OpenGL framework used to render in two dimensions (2D) andthree dimensions (3D) in a graphic content on a display), databaselibraries (e.g., SQLite to provide various relational databasefunctions), web libraries (e.g., WebKit to provide web browsingfunctionality), and the like. The libraries 1010 can also include a widevariety of other libraries 1028 to provide many other APIs to theapplications 1006.

The frameworks 1008 provide a high-level common infrastructure that isused by the applications 1006. For example, the frameworks 1008 providevarious graphical user interface (GUI) functions, high-level resourcemanagement, and high-level location services. The frameworks 1008 canprovide a broad spectrum of other APIs that can be used by theapplications 1006, some of which may be specific to a particularoperating system or platform.

In an example, the applications 1006 may include a home application1036, a contacts application 1030, a browser application 1032, a bookreader application 1034, a location application 1042, a mediaapplication 1044, a messaging application 1046, a game application 1048,and a broad assortment of other applications such as a third-partyapplication 1040. The applications 1006 are programs that executefunctions defined in the programs. Various programming languages can beemployed to produce one or more of the applications 1006, structured ina variety of manners, such as object-oriented programming languages(e.g., Objective-C, Java, or C++) or procedural programming languages(e.g., C or assembly language). In a specific example, the third-partyapplication 1040 (e.g., an application developed using the ANDROID™ orIOS™ software development kit (SDK) by an entity other than the vendorof the particular platform) may be mobile software running on a mobileoperating system such as IOS™, ANDROID™, WINDOWS® Phone, or anothermobile operating system. In this example, the third-party application1040 can invoke the API calls 1050 provided by the operating system 1012to facilitate functionality described herein.

The terms and expressions used herein are understood to have theordinary meaning as is accorded to such terms and expressions withrespect to their corresponding respective areas of inquiry and studyexcept where specific meanings have otherwise been set forth herein.Relational terms such as first and second and the like may be usedsolely to distinguish one entity or action from another withoutnecessarily requiring or implying any actual such relationship or orderbetween such entities or actions. The terms “comprises,” “comprising,”“includes,” “including,” or any other variation thereof, are intended tocover a non-exclusive inclusion, such that a process, method, article,or apparatus that comprises or includes a list of elements or steps doesnot include only those elements or steps but may include other elementsor steps not expressly listed or inherent to such process, method,article, or apparatus. An element preceded by “a” or “an” does not,without further constraints, preclude the existence of additionalidentical elements in the process, method, article, or apparatus thatcomprises the element.

In addition, in the foregoing Detailed Description, it can be seen thatvarious features are grouped together in various examples for thepurpose of streamlining the disclosure. This method of disclosure is notto be interpreted as reflecting an intention that the claimed examplesrequire more features than are expressly recited in each claim. Rather,as the following claims reflect, the subject matter to be protected liesin less than all features of any single disclosed example. Thus, thefollowing claims are hereby incorporated into the Detailed Description,with each claim standing on its own as a separately claimed subjectmatter.

The examples illustrated herein are described in sufficient detail toenable those skilled in the art to practice the teachings disclosed.Other examples may be used and derived therefrom, such that structuraland logical substitutions and changes may be made without departing fromthe scope of this disclosure. The Detailed Description, therefore, isnot to be taken in a limiting sense, and the scope of various examplesis defined only by the appended claims, along with the full range ofequivalents to which such claims are entitled.

Additional objects, advantages and novel features of the examples willbe set forth in part in the description which follows, and in part willbecome apparent to those skilled in the art upon examination of thefollowing and the accompanying drawings or may be learned by productionor operation of the examples. The objects and advantages of the presentsubject matter may be realized and attained by means of themethodologies, instrumentalities and combinations particularly pointedout in the appended claims.

What is claimed is:
 1. An eyewear device comprising: a supportstructure; a first input surface on a first side of the supportstructure; an image display supported by the support structure andpositioned to present an image to a user; an image display drivercoupled to the image display to control the image presented to the user;a memory; a processor coupled to the first input surface, the imagedisplay driver, and the memory; and programming in the memory, whereinexecution of the programming by the processor configures the eyeweardevice to perform functions, including functions to: receive an imagefrom another user within an application executing on the eyewear device;in response to receiving the image from another user, present a firstgraphical user interface (GUI) on the image display including availableactions for responding to the another user in response to the image fromthe another user within the application; detect a first finger touch onthe first input surface; select an available action in accordance withthe detected first finger touch on the first input surface; and inresponse to selecting the available action, present a second GUI on theimage display based on a first visual perspective of the first inputsurface, the second GUI including additional available actions forselection by a second finger touch on the first input surface to respondto the another user in response to the image from the another user. 2.The eyewear device of claim 1, wherein execution of the programming bythe processor further configures the eyewear device to: identify afinger gesture on the first input surface of the eyewear device bydetecting at least one touch event; and adjust the first GUI presentedon the image display of the eyewear device responsive to the identifiedfinger gesture.
 3. The eyewear device of claim 2, wherein the first GUIincludes at least one symbol associated with the identified fingergesture and wherein the at least one symbol is highlighted in responseto the identified finger gesture.
 4. The eyewear device of claim 1,wherein the processor is further configured to: monitor the applicationexecuting on the eyewear device; determine a current state of themonitored application; present a plurality of GUIs based on thedetermined current state of the monitored application; and select thefirst GUI from the plurality of GUIs responsive to the determined stateof the monitored application.
 5. The eyewear device of claim 1, whereinthe first GUI includes a plurality of selectable actions, eachselectable action corresponding to one of a plurality of non-tap fingergestures, one of the selectable actions is designated as a defaultaction, and the processor is further configured to: identify a fingertap gesture on the first input surface; and perform the default actionin response to the finger tap gesture.
 6. The eyewear device of claim 5,wherein execution of the programming by the processor further configuresthe eyewear device to: identify a finger hold gesture on the first inputsurface; and display the first GUI until the finger is removed from thefirst input surface without performing the default action.
 7. Theeyewear device of claim 1, wherein the detection includes detecting aposition of the first finger touch on the first input surface andwherein the first GUI displays a first virtual input surfacecorresponding to the first input surface and a virtual pointer adjacentthe first virtual input surface in a virtual position corresponding tothe position of the first finger touch on the first input surface andpositioned to correspond to the first finger touch on the first inputsurface.
 8. The eyewear device of claim 1, wherein the device furthercomprises: a second input surface on a second side of the supportstructure; and wherein execution of the programming by the processorfurther configures the eyewear device to: detect a second finger touchon the second input surface; and present a third graphical userinterface (GUI) on the image display responsive to the detected secondfinger touch on the second input surface.
 9. The eyewear device of claim8, wherein the image display driver presents the image on the imagedisplay as a three-dimensional image and wherein the processorconfigures the eyewear device to present the first GUI or the second GUIon the image display from the first visual perspective of the firstinput surface in response to the first finger touch and to transition todisplaying the third GUI on the image display from a second visualperspective of the second input surface in response to the second fingertouch on the second input surface.
 10. The eyewear device of claim 9,further comprising: a remote accessory configured for wirelesscommunication with the processor; and wherein execution of theprogramming by the processor further configures the eyewear device to:detect an activation signal from the remote accessory; present a fourthgraphical user interface (GUI) on the image display from a third visualperspective of the remote accessory in response to the detectedactivation signal from the remote accessory; and adjust the fourth GUIpresented on the image display of the eyewear device in response toinputs from the remote accessory.
 11. A method for controlling aneyewear device of a user, the method comprising: receiving an image fromanother user within an application executing on the eyewear device; inresponse to receiving the image from another user, presenting a firstgraphical user interface (GUI) on an image display including availableactions for responding to the another user in response to the image fromthe another user within the application; detecting a first finger touchon a first input surface on one side of an eyewear device; selecting, onthe image display of the eyewear device, an available action inaccordance with the detected first finger touch on the first inputsurface for controlling the eyewear device; and in response to selectingthe available action, present a second GUI on the image display based ona first visual perspective of the first input surface the second GUIincluding additional available actions for selection by a second fingertouch on the first input surface to respond to the another user inresponse to the image from the another user.
 12. The method of claim 11,further comprising: identifying a finger gesture on the first inputsurface of the eyewear device by detecting at least one touch event; andadjusting the first GUI presented on the image display of the eyeweardevice responsive to the identified finger gesture.
 13. The method ofclaim 12, wherein the first GUI includes at least one symbol associatedwith the identified finger gesture, and the adjusting compriseshighlighting the at least one symbol in response to the identifiedfinger gesture.
 14. The method of claim 11, further comprising:monitoring the application executing on the eyewear device; determininga current state of the monitored application; presenting a plurality ofGUIs based on the determined state of the monitored application; andselecting the first GUI from the plurality of GUIs responsive to thedetermined state of the monitored application.
 15. The method of claim11, wherein the first GUI includes a plurality of selectable actions,each selectable action corresponding to one of a plurality of non-tapfinger gestures, one of the selectable actions is designated as adefault action, and the method further comprises: identifying a fingertap gesture on the first input surface; and performing the defaultaction in response to the finger tap gesture.
 16. The method of claim15, further comprising: identify a finger hold gesture on the firstinput surface; and display the first GUI until the finger is removedfrom the first input surface without performing the default action. 17.The method of claim 11, wherein the detecting comprises detecting aposition of the first finger touch on the first input surface andwherein the first GUI displays a first virtual input surfacecorresponding to the first input surface and a virtual pointer adjacentthe first virtual input surface in a virtual position corresponding tothe position of the first finger touch on the first input surface andpositioned to correspond to the first finger touch on the first inputsurface.
 18. The method of claim 11, further comprising: detecting asecond finger touch on a second input surface on another side of theeyewear device; and presenting a third graphical user interface (GUI) onthe image display responsive to the detected second finger touch on thesecond input surface.
 19. The method of claim 18, further comprising:presenting the first GUI or the second GUI on the image display from thefirst visual perspective of the first input surface in response to thefirst finger touch; presenting the third GUI on the image display from asecond visual perspective of the second input surface in response to thesecond finger touch on the second input surface; detecting an activationsignal from a remote accessory; presenting a fourth graphical userinterface (GUI) on the image display from a third visual perspective ofthe remote accessory in response to the detected activation signal fromthe remote accessory; and adjusting the fourth GUI presented on theimage display of the eyewear device in response to inputs from theremote accessory.
 20. A non-transitory computer-readable medium storingprogram code which, when executed, is operative to cause at least oneelectronic processor of an eyewear device of a user to perform the stepsof: receiving an image from another user within an application executingon the eyewear device; in response to receiving the image from theanother user, presenting a first graphical user interface (GUI) on animage display including available actions for responding to the anotheruser in response to the image from the another user within theapplication; detecting a first finger touch on a first input surface onone side of an eyewear device; selecting, on the image display of theeyewear device, an available action in accordance with the detectedfirst finger touch on the first input surface for controlling theeyewear device; and in response to selecting the available action,present a second GUI on the image display based on a visual perspectiveof the first input surface, the second GUI including additionalavailable actions for selection by a second finger touch on the firstinput surface to respond for response to the another user in response tothe image from the another user.